Skip to content

[GRDM-63966] waterbutler の依存先を RDM-waterbutler に変更し、共有する依存を更新(第2段階) - #10

Open
miyamoto83 wants to merge 231 commits into
RCOSDP:developfrom
miya-biz:feature/vuln-remediation-rdm-wb
Open

miyamoto83 wants to merge 231 commits into
RCOSDP:developfrom
miya-biz:feature/vuln-remediation-rdm-wb

Conversation

@miyamoto83

Copy link
Copy Markdown

Ticket

GRDM-63966

Purpose

第 1 段階(#9)に続く第 2 段階。waterbutler の依存先を CenterForOpenScience/waterbutler 26.6.1 から RCOSDP/RDM-waterbutler の Python 3.13 対応ブランチに変更し、共有する依存をその固定版に揃えて、第 1 段階で残していた脆弱性を解消する。

本 PR は #9 のブランチの直上に 2 コミットを追加したもので、差分の大半は #9 と重複する。#9 が develop に入ると、本 PR の差分はその 2 コミット分(pyproject.toml、poetry.lock)になる。

Changes

  • waterbutler の依存先を RCOSDP/RDM-waterbutler feature/vuln-remediation(926e21f8)に変更した。pyproject.toml は pull request ref(refs/pull/101/head)を指し、poetry.lock で 926e21f8 に固定している(13a19b5)。
  • 共有する依存を RDM-waterbutler の固定版に揃えた: aiohttp 3.14 系、tornado 6.6、celery 5.6.3、sentry-sdk 2.73.0、newrelic 14.1.0、stevedore 5.9.1、google-auth 2.62.0、certifi 2026.10.23、setuptools 84 系。waterbutler 経由の PyJWT、cryptography、oauthlib、aiobotocore、boto3 / botocore も RDM-waterbutler の版になり、RDM-waterbutler が使う Swift、Keystone、Azure Storage のクライアントライブラリが lock に加わった(13a19b5)。
  • requests を 2.34.2 に更新した。RDM-waterbutler の依存として実行イメージに入るようになったため(3b9b001)。
  • 第 1 段階で CenterForOpenScience/waterbutler 26.6.1 の固定版に由来して残っていた脆弱性(aiohttp、tornado、PyJWT、cryptography、oauthlib、certifi、setuptools)を解消した。

Side effects

  • MFR のコードは変更していない。動作の変更はない。
  • 第 1 段階で設定したリクエスト本文(64KB)とヘッダ(32KB)の上限は、tornado 更新後も入力制限として維持する。

QA Notes

  • CI: 実行 43172130284(ユニットテスト 248 件、flake8、unoserver 結合テスト、mfr と unoserver 両イメージの Trivy 検査を含む)
  • Trivy(DB 2026-11-12): 実行イメージ 1,332 件、CRITICAL 18 件、修正版が提供されているものは 1 件。第 1 段階(同一 DB)は 1,420 件、CRITICAL 20 件、修正版あり 89 件で、その 89 件はすべて解消した。残る修正版ありの 1 件は oauthlib 4.0.0 で、RDM-waterbutler が完全一致で固定する依存である。MFR・WaterButler のコードは参照しておらず、OAuth サーバー側のデバイス認可フローの問題のため到達経路はない。RDM-waterbutler 側の固定を上げた時点で lock を更新して取り込む。それ以外の検出は第 1 段階と同じである。
  • E2E: miya-biz/RDM-e2e-test-nb feature/mfr-python313(282bffb)で、本ブランチと miya-biz/RDM-waterbutler feature/vuln-remediation(926e21f8)からビルドしたイメージを指定して実行。実行 43203342963 は失敗ジョブの再実行後に、WEKO 以外の 12 ジョブ中 9 成功。ファイルプレビューを含む User Tests は成功。失敗 3 件(Metadata、Wiki、Migration 20250906)は MFR / WaterButler を経由しない。

Deployment Notes

felliott and others added 30 commits July 1, 2020 15:31
 * Add four new switches/envvars to manage keen logging:

   * `KEEN_ENABLE_LOGGING` will turn on/off *all* keen logging.

   * `KEEN_PRIVATE_LOG_VIEWS` will set logging for views to the
     private collection.

   * `KEEN_PRIVATE_LOG_ERRORS` will set logging for errors to the
     private collection.

   * `KEEN_PUBLIC_LOG_VIEWS` will set logging for views to the public
     collection.
* The python module `certifi` was pinned to a very old version to
   work around issues with Rackspace. Not only is it no longer
   relevant, it can no longer successfully validate modern TLS certs
 * conversion based off @aaxelb's work in @CenterForOpenScience/SHARE

 * update CI badge urls in README.md

 * add explicit upload to coveralls

   * replace python-coveralls with coveralls in dev-requirements.txt

 * remove old travis config files

 [ENG-3265]
 * Allow the parent window of the MFR iframe to pass its own url when
   starting the hypothes.is annotator. Parents using `postMessage`
   must now send an object with two keys: `type: 'startHypothesis'`
   and `parentUrl: <parent window url>`. Parents using pym must now
   call `.sendMessage('setParentUrl', <parent window url>)`. The
   included pym code has been updated to do this.

   MFR overrides the annotator to pass the parent window's url rather
   than its own, b/c the parent window url is more stable and
   descriptive. It was previously doing this by calling
   `document.referrer` from within the iframe. Due to privacy
   concerns, browsers have started allowing the iframe to only see the
   host and not the path of the parent url. Parent frames must now
   pass in their own url to get around this.
 * Upgrade Dockerfile image base to python v3.6.

 * Install gosu via apt-get. gosu v1.10 started being shipped with
   debian as of the `stretch` release.  Install from apt-get instead
   of `curl`-ing it from the github release page.

 * Pin pip version to 18.1 to avoid dep conflicts.

 * Based off the following WB commits:
   * CenterForOpenScience/waterbutler@d03a84c
   * CenterForOpenScience/waterbutler@b179eb8
 * Bump reportlab dep to v3.6.5 to unbreak `invoke install` on macOS.
 * add boa to supported extensions

 * bump pygments dep to 2.12 to support boa

 * add debugging logs to codepygments extension
felliott and others added 29 commits October 5, 2026 22:54
… the Debian package instead of fetching it with keyserver verification, and exclude the report script from flake8 because it requires Python 3.6+
…/vuln-remediation, keeping the RDM CJK CMap files, the allow-modals iframe sandbox and the README lines unrelated to the Python 3.13 update
… runtime stages, move LibreOffice into a separate unoserver image (Dockerfile.unoserver) built on Ubuntu 26.04 because Debian 13 links LibreOffice against libxml2 2.9.14 whose open CVEs cannot be fixed there (the fixed 2.15 changed ABI) and without the unused pebble binary the Ubuntu base image ships, apply security updates, copy the nbconvert templates into the runtime image and verify the dependency tree with pip check
…age image and scan both the mfr and unoserver images with Trivy
…dt/ods with Japanese text) and run them in CI against the unoserver image
…cies: Pillow 13.0.0, mistune, Mako, nbconvert 7.17.2, psd-tools, Jinja2, Markdown, Pygments and ipython to versions with security fixes, propcache 0.5.4 (0.4.0 was yanked for a reference leak) and the unpinned transitive libraries (urllib3, pyasn1, soupsieve, bleach, idna, multidict) to their current releases; move black/isort/ruff to the dev group, port PSD export to PSDImage.open().composite() and drop the stray scikit-image test import
… of the original URL and keep the response open for streaming; add provider download tests
…Keen analytics when no project id is configured; add HTTP provider tests
…ar) and close it on shutdown instead of leaking a session per request
…s with asyncio.sleep so the event loop keeps serving requests, and give the worker its own control exchange so it does not receive pickle control messages from other applications sharing the broker
…andlers only answer GET, but tornado parses form bodies before the method check
…nstead of the rpc backend, whose shared reply queue leaves results of earlier tasks unacknowledged until RabbitMQ closes the channel and the next task call fails; pin the default in a test
… 1.10.1 and prototype HTML pages under the pdf extension, and the GLmol/three.js files and the foundation/jquery-ui/qunit stylesheets under the pdb extension, none of which any template loads
… out since the extension was added in 2015, so .svg files are converted through unoconv and the extension's template and bundled SVGInjector 1.1.3-RC are never served
… Bootstrap 3.3.5 and jquery-mousewheel 3.1.13 in package.json, copy the served files from node_modules into the static directories with `invoke assets` during the Docker build instead of keeping copies in the repository (the server copy of the Bootstrap stylesheet was 3.3.4 and now matches the 3.3.5 script), and test that every file the templates load exists
…2.4.45 from npm together with the jquery.event.drag plugin it ships (the bundled 2.2 copy does not load under jQuery 3); replace the image viewer's removed load() event shorthand with on("load"), and check that the files the stylesheets refer to exist
…m npm (jQuery Zoom, pv with its gl-matrix build and JSC3D) in a CycloneDX SBOM so image scans see them, and test that the paths it records exist
…on 2 and invoke problems it describes cannot occur with the Poetry-based setup
…ull/101/head, 926e21f8) instead of the COS release tag and align the shared pins with it (aiohttp ^3.14, tornado 6.6, celery 5.6.3, sentry-sdk 2.73.0, newrelic 14.1.0, stevedore 5.9.1, google-auth 2.62.0, certifi 2026.10.23, setuptools ^84)
@miyamoto83
miyamoto83 marked this pull request as ready for review November 13, 2026 03:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.